Skip to content

[SPARK-56186][PYTHON] Retire pypy#54988

Closed
gaogaotiantian wants to merge 1 commit intoapache:masterfrom
gaogaotiantian:retire-pypy
Closed

[SPARK-56186][PYTHON] Retire pypy#54988
gaogaotiantian wants to merge 1 commit intoapache:masterfrom
gaogaotiantian:retire-pypy

Conversation

@gaogaotiantian
Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

We retire pypy

  • Remove all pypy related code in pyspark (actually the only mattered one is for simple traceback so it probably will still work)
  • Remove all pypy skips for tests
  • Remove master CI for pypy. branch-4.0 and branch-4.1 tests are kept
  • Remove pypy 3.11 docker image (3.10 is kept for testing)
  • Remove pypy from docs (we should probably do it for the actual spark website too)

Why are the changes needed?

We had a discussion in https://lists.apache.org/thread/glcq0zgr33sozo7y4y7jqph24yh3m92p about dropping support for pypy and we have many +1s and no -1s.

numpy dropped support for pypy and pypy is not really in active development.

Does this PR introduce any user-facing change?

Yes, we don't officially support pypy anymore. We still expect most of the old pypy code to work but we should not make any promises.

How was this patch tested?

CI.

Was this patch authored or co-authored using generative AI tooling?

No.

Copy link
Copy Markdown
Member

@HyukjinKwon HyukjinKwon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM if it works

@HyukjinKwon
Copy link
Copy Markdown
Member

Merged to master.

@gaogaotiantian gaogaotiantian deleted the retire-pypy branch March 25, 2026 22:49
hadoop: hadoop3
envs: >-
{
"PYSPARK_IMAGE_TO_TEST": "pypy-310",
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since we remove the pypy3.10 workflow file, I think we don't need to keep its docker file? @gaogaotiantian

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The docker file was kept to run 4.0/4.1 pypy tests (it was used there right?). We can drop support for pypy from 4.2 but we can't drop it for previous versions.

Copy link
Copy Markdown
Contributor

@zhengruifeng zhengruifeng Mar 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think removing it from master branch won't affect 4.0/4.1, since we use the branchs as the tags:
e.g. in https://github.com/apache/spark/actions/runs/23605397532/job/68748707119

the image is

Starting job container
  /usr/bin/docker --config /home/runner/work/_temp/.docker_e68d0502-441d-4d67-9d06-ff417888c8b7 login ghcr.io -u dongjoon-hyun --password-stdin
  /usr/bin/docker --config /home/runner/work/_temp/.docker_e68d0502-441d-4d67-9d06-ff417888c8b7 pull ghcr.io/apache/apache-spark-ci-image-pyspark-pypy-310:branch-4.1-23605397532

with tag branch-4.1-23605397532

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants